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November 2007 

New research from the Center for Collaborative Education finds that students in 
Boston’s Pilot high schools outperform students from other non-exam Boston 
Public Schools on every standard measure of engagement and performance. 

This level of achievement holds for every racial, economic, and academic 
subgroup examined. Pilot high school students show better MCAS scores, 
higher attendance rates, higher promotion rates — and the four-year graduation 
rate for 2006 was more than 23 percentage points higher than the rate for BPS 
students, 75.7% as compared with 52.2% for BPS. (“BPS” in this report refers to 
non-Pilot, non-exam schools.) 

The study found that Pilot high schools reflect the BPS demographics in 
terms of race, income, and mainstream special needs students. The report 
also identifies some areas in which Pilot School demographics fall short of 
the goal of representing the student population of the Boston school district. 
Pilot high schools have proportionately fewer students designated as Limited 
English Proficient and fewer students with moderate to severe special needs. 

In addition, proportionately fewer students 
arrive with certain other warning signs of “risk,” 
such as low grade 8 math MCAS scores and 
poor attendance records in eighth grade. It is 
not possible to isolate how much differences in 
populations, in addition to the Pilot features 
of schools, affected differences in performance 
outcomes. 

However, the news from the report is that Pilot high school students in every 
category — including students with risk factors — performed better than their 
counterparts in the Boston school district. 

The present study offers a review of Pilot high school student performance over 
the course of four years, looking at overall outcomes. It then parses the data 
to identify how different populations of students are being served by the Pilot 
Schools. 

Defining Pilot Schools 

Pilot Schools were first created in Boston in 1995 through a unique partnership 
that included the mayor, the office of the school superintendent, the school 
committee and the teachers union. An essential characteristic of Pilot Schools is 
that they are freed from district mandates and union work rules to have greater 
control over budget, staffing, curriculum, governance, and schedule in order 
to provide better education for their students. Pilots are designed to serve the 
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same student population as the district schools, and have several additional 
defining characteristics: 

• Pilots are accountable: Pilot Schools and the district have 
developed a set of consistent benchmarks, against which 
performance is evaluated every five years. 

• Pilots are small and personalized: Pilot Schools place great 
emphasis on creating a nurturing school culture in which teachers 
can attend closely to each student’s learning needs. 

• Pilots are vision driven: Every Pilot School has created a vision 
focused on equity and the fundamental belief in each child’s 
potential. Pilot Schools have the power to hire teachers and staff to 
support the culture and vision of each individual school. 

A Summary of the findings 

Among the highlights of the report’s findings are results that demonstrate the 
power of the Pilot School model. These include the following: 

• Pilot high schools serve students well on 
every outcome in a wide range of measurable 
performance and engagement criteria. 

• The four-year Pilot high school graduation rate 
for the most recent year was 23 percentage 
points higher for Pilots (75.7%) than for BPS 
(52.2%). 

• Pilot high schools have attracted students that 
represent the district’s racial, economic, and 
mainstream special education subgroups. 

• Pilot high schools enroll a lower percentage of 
students with risk factors, independent of the 
student assignment process, suggesting that a 
disproportionate number of students without 
risk factors who are seeking high-performing 
schools choose to apply to Pilot Schools. 

• Students with risk factors perform better in 
Pilot high schools than in district schools. 

• Comparisons of the MCAS scores of students in 
Pilot high schools with those of like students in 
all subgroups in district schools show stronger 
performance among Pilot high school students. 



Taken together, 
the student engage- 
ment and 
performance find- 
ings show that Pilot 
high school students 
are outperforming 
the district average 
on a range of 
indicators of 
behavior and 
academics. 



Scope of this study 

This is the most comprehensive examination of Pilot high school performance 
to date, using data provided by the Boston School Department to examine 
Boston’s closely watched Pilot high schools over a four-year period (2001-05). 
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In addition to aggregate analyses, the report breaks down the numbers to 
discover who is attending Pilot high schools and how subgroups of the Boston 
student population are faring in Pilot high schools. 

Looking at demographic sub-groups 

The report finds that Pilot high school students as a group are doing 
substantially better on all measures of academic success than their peers in 
the district. It then breaks down the larger group into subgroups by racial, 
economic, and academic designations to answer two important questions: 

• How comparable is Boston’s Pilot high school population to the 
district high school population? 

• How well do the specific subgroups fare in Pilot high schools in 
comparison to similar groupings in the district as a whole? 

The study looks at grade 10 MCAS results for the three larger racial groups — 
Black, Latino, and White — and finds that for each group over four years the 
pass rates are substantially higher for Pilot School students than BPS students 
in both math and English language arts (ELA). Black and Latino Pilot students 
had higher pass rates in seven out of the eight administered tests, while White 
Pilot students passed at higher rates in all eight tests. 

The achievement gaps — between White and Black students, and between White 
and Latino students — as measured by MCAS scores continued to be a problem 
within both Pilot and BPS schools in all four years, on both ELA and math tests. 
However, there is promising news for Pilot Schools: at the beginning of the 
period tracked in the study, the gaps were greater in the Pilot Schools; at the 
end they were smaller in Pilots than in BPS schools. 

Looking at economic variables, the study finds comparable enrollment in Pilot 

and BPS schools: 68.4% of Pilot 
students and 69.8% of BPS students 
were eligible for free/reduced- 
price lunch, a common proxy for 
low-income students. In each year 
and on both grade 10 MCAS tests, 
low-income Pilot School students 
passed at a higher rate than their 
BPS counterparts — by more than 
20 percentage points on five of the 
eight tests. 

The study examines the academic history of Pilot students to see whether 
students with risk factors — namely, low eighth-grade attendance rates; having 
received a warning grade on their grade 8 math MCAS exam; or being over-age 
first-time ninth graders — are represented proportionately to BPS schools, and 
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how such students fare in Pilot high schools. 
The data show that, while there are substantial 
numbers of students with risk factors entering 
Pilot high schools, the proportion is lower than 
in BPS schools. However, Pilot ninth graders 
who received a warning on their grade 8 math 
MCAS performed substantially better than 
corresponding BPS ninth graders. They had 
higher ninth-grade attendance rates, higher 
promotion rates to tenth grade, and higher 
passing rates on both grade 10 MCAS exams. 




The study also finds that, while Pilot high schools serve similar percentages 
of mainstream special education students to BPS schools, they serve 
proportionately fewer students designated as Limited English Proficient and 
students who have moderate to severe special needs, although the gap has been 
closing in recent years for the special needs population. While the numbers for 
these Pilot populations were too small to analyze comparative results, the study 
found that the aggregate differences in MCAS pass rates between Pilot and 
BPS schools were not affected by the disproportionate representation of these 
students. 

Impact of admissions policies on Pilot high school 

ENROLLMENT 

The ten Pilot high schools have a variety of admissions processes that often 
differ from BPS schools. Two admit strictly by lottery (like district schools); 
two accept only over-age students; one is a performing arts school with 
academically blind auditions; one is a pathway school that admits students from 
its feeder school; one is a Horace Mann School with a state-mandated lottery 
and application process; and three ask students to complete an application to 
evaluate the match between student and school. 

The study finds that Pilot Schools with 
different kinds of admissions processes, 
including both application and lottery 
processes, end up with a lower percentage 
of students with risk factors than are found 
in BPS schools. Even though in the first year 
studied lottery schools had students with 
some risk factors in proportions that were 
equivalent to BPS schools, as demand shifted, 
the percentages of students with risk factors 
decreased each succeeding year. The study 




www.cce.org Strong Results, High Demand: A Four-Year Study of Boston's Pilot High Schools 




vii 




Executive 

Summary 





Executive 

Summary 



viii 






concludes that good schools attract the interest of a disproportionately high 
number of college-bound students, which would suggest a correspondingly 
smaller proportion of students with risk factors. 

Key policy recommendations 

Analysis of the data in this four-year study supports the following courses of 
action: 

• Create more Pilot Schools. Pilot Schools are more effective for 
students from all backgrounds and of all levels of achievement and are 
being chosen by more students than they can currently serve. While 
the Boston Public Schools’ student population has been declining, the 
demand for enrollment in Pilot high schools has remained beyond 
capacity. 

• Revisit and strengthen the Boston Public Schools student 
assignment process to encourage informed choices of high 
schools by all students. Students need information about the schools 
they are choosing so they can make informed, intentional choices. 
Informed students will lead to increased positive matches between 
students and schools. A likely outcome will be increased engagement and 
performance across the district’s high schools. 

• Enroll representative proportions of students who are 
designated Limited English Proficient, as well as students with 
moderate to severe special needs, in Pilot Schools. As much as 
possible, these schools should represent the BPS population. 

• Share the lessons of Pilot Schools. This has always been the intent of 
the Pilot program — to help Boston improve education throughout the 
district. 




As district public schools that were created 
to serve as laboratories of innovation, 

Pilot high schools are in high demand and 
have strong results. They are achieving the 
goals of equity and excellence within the 
public school district. The experiment in 
innovation that the Boston Public Schools 
and the Boston Teachers Union created 12 
years ago has demonstrated results that 

suggest that the Boston Public Schools need look no further than their own 
Pilot Schools for examples of high-performing high schools. Families across 
race and income lines, and students with risk factors all seek high-performing 
schools. The logic presented in the creation of Boston’s Pilot Schools in 1995 
still holds. Demand for quality high schools exceeds supply. The challenge of the 
district and the teachers union is to respond by working together to create more 
quality school choices for Boston families. 
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Introduction 

Until 2006, Boston Public Schools was the only district in the country with Pilot 
Schools, which were granted autonomy over school design and were created to serve as 
research and development sites for the district. Because Pilot Schools have innovative 
practices and have demonstrated a capacity to improve student learning, the public 
and the education community look with interest upon these schools as models for 
urban school reform. For the first time, available data allow an in-depth longitudinal 
examination of Pilot School student performance. Because of the recent national and 
district attention to high school reform, this paper focuses on Boston’s ten Pilot high 
schools. Through examination of multiple subgroups and cohorts of students from 2001 
through 2005, the report seeks to understand the enrollment patterns and outcomes of 
Boston students in Pilot high schools in comparison to other district high schools. 

Boston Pilot Schools, currently a network of 20 schools within the Boston Public School 
district, were created in 1994 through an agreement among the city’s mayor, school 
committee, and teachers union. 1 At that time, the district was concerned about the 
potential loss of Boston students to newly opening charter schools, and proactively 
created the Pilot model. Through this agreement, Pilot Schools were granted autonomy 
over five key areas of school design and operations: 

• Budget 

• Staffing 

• Governance 

• Curriculum, instruction, and assessment 

• Schedule 

In addition to being autonomous, Pilot Schools place high priority on being: 

• Accountable: Pilot Schools are held to high standards of performance through 
a high-stakes school quality review process every five years, using a set of 
benchmarks that articulate the criteria for high-performing schools. 

• Small: Pilot Schools enroll 450 students or fewer, enabling adults to know 
students well. Every school places great emphasis on creating a nurturing school 
culture in which staff pay close attention to each student’s learning needs. 

• Vision driven: Every Pilot School has an articulated vision of educating all of its 
students, with teaching and learning at the vision’s core. Pilot Schools have the 
latitude to hire staff members who are committed to fulfilling the school’s vision. 

• Focused on equity: Pilot Schools embrace as a core belief the potential of every 
student to achieve academic success and graduate from college, regardless of his 
or her background and past educational experience. 

Pilot Schools belong to a Network convened by the Center for Collaborative Education, 
a nonprofit education organization that provides coaching, professional development, 



x Two of the 20 — Boston Day and Evening Academy and Health Careers Academy — are Horace Mann charter schools in 
addition to being part of the Pilot School Network. Horace Mann charters are granted autonomy by the state department 
of education while also remaining part of the district and the teachers union. 
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advocacy, and research for the schools. Together, they commit to Network principles 
of high expectations, personalized teaching and learning, and family involvement 
(Appendix 1). 

Since their inception, Pilot Schools have received increasing attention for their 
innovative practices and strong outcomes (CCE, 2001a; CCE, 2004a; Tung, Ouimette, 
and Rugen, 2006). This paper focuses on the ten Boston Pilot and Horace Mann high 
schools within the Network, which served about 13.5% of Boston public high school 
students in 2004-05. The ten Pilot high schools are diverse by mission, theme, and 
instructional focus. They are listed in the following table (additional information in 
Appendix 2). 

Table 1: Boston's Pilot High Schools 



School Name 


Grades 

Served 


Enrollment 
in 2005 


How Became 
Pilot 


School Focus 


Another Course to College 


9 to 12 


248 


Conversion 
from a 
program 


College prep 


Boston Arts Academy 


9 to 12 


406 


Start-up 


College prep, visual 
and performing arts 
focus 


Boston Community 
Leadership Academy 


9 to 12 


442 


Conversion 
from regular 
status 


College prep, 
leadership focus 


Boston Day and Evening 
Academy 

(Horace Mann Charter) 


ungraded 


392 


Start-up 


Over-age, 
college prep 


Fenway High School 


9 to 12 


275 


Conversion 
from a 
program 


College prep 


Greater Egleston 
Community High School 


10 to 12 


104 


Conversion 
from a 
program 


Over-age, 
college prep 


Health Careers Academy 
(Horace Mann Charter) 


9 to 12 


209 


Start-up 


College prep, 
health focus 


Josiah Quincy Upper 
School 


6 to 12 


433 


Start-up 


College prep, 
feeder K-5 school 


New Mission High School 


9 to 12 


244 


Start-up 


College prep 


TechBoston Academy 


9 to 12 


236 


Start-up 


College prep, 
technology focus 



Subsequent to the creation of Pilot Schools, the Boston Public School (BPS) district 
has instituted several other significant changes in the choices offered to Boston high- 
school-age residents. In 2001, in addition to the Pilot high schools, BPS had 9 large, 
comprehensive high schools, 3 exam schools, 1 career and technical school, and 2 small 
schools. Over the course of several years, 4 of the 9 large, comprehensive high schools 
have been transformed into 12 small schools in educational complexes. The remaining 
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5 large, comprehensive high schools each contain multiple small learning communities. 
Four other large high schools remain: 3 examination schools and 1 career and technical 
education school. Thus, BPS’s high school reform strategy evolved to embrace a 
“managed portfolio” approach that allows students and families to choose from the 
following different types of public high schools: 

• Large, comprehensive high school with small learning communities (5) 

• Examination school (3) 

• Career and technical education school (1) 

• Small school housed in education complex (12) 

• Small free-standing school (2) 

• Pilot School (10) 

School choice has been a policy that many urban districts have adopted in recent years 
with mixed support and mixed results (Goldhaber, 1999; Powers and Cookson, 1999). 
Given that Boston high school students have the multiple public school choice options 
listed above, the public and the district may ask about the outcomes and consequences 
of the choice policy: Does choice lead to better educational outcomes for students, and if so, 
for which school types? 

With the recent changes in high school offerings and increased availability of student- 
level data, questions about Pilot high school outcomes in engagement and performance 
in comparison with the district’s non-Pilot schools may be answered in more depth 
than in the past. This paper expands on results reported in an earlier one-year study 
(Tung, Ouimette, and Rugen, 2006), using a four-year, student-level database (2001-02 
to 2004-05) to understand outcomes across Boston’s high schools. Specifically, the 
following research questions guide an analysis of educational outcomes and student 
enrollment in Pilot high schools as compared with other BPS high schools: 

I. How do Pilot high school students perform in comparison 
with other district high schools across a range of 
engagement and performance indicators? 

II. Whom do Pilot high schools serve? 

III. How are students with risk factors performing in Pilot 
high schools? 

IV. How are students in different demographic subgroups 
performing in Pilot high schools? 

V. How does lottery assignment or application for admission 
affect Pilot high school enrollment? 
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The data analyzed in this paper were obtained from the Boston Public Schools (BPS) 
and represent all students enrolled at any point in time in BPS high schools, grades 
9-12, for each of the four school years: 2001-02, 2002-03, 2003-04, and 2004-05. 

The indicators that BPS provided to the Center for Collaborative Education (CCE) 
are the same that it provides to the Massachusetts Department of Education’s 
Student Information Management System (SIMS) database. BPS also provided the 
Massachusetts Comprehensive Assessment System (MCAS) results that were obtained 
from the Massachusetts Department of Education. 

All data were received at the individual student level. All four years of SIMS and MCAS 
data files were cleaned of duplicate lines of data, inconsistent data was reconciled and 
new variables were created as needed, without loss of any raw data. Each year’s files 
were then merged into one database. Analyses were performed using both SPSS and 
Excel. Each analysis excluded students who were enrolled in the district for only one day 
and/or attended zero days in a year as well as students attending schools serving special 
populations. 2 Students were aggregated into three school types: 

• Boston Public Schools (BPS): all high school students enrolled in Boston 
Public Schools who attended schools that did not have Pilot or exam status. 

• BPS Pilot high schools: all high school students enrolled in Boston Public 
Schools that have been granted Pilot status by the district. 

• Exam schools: all high school students enrolled in the three Boston high 
schools that admit students based on entrance examination results and 
prior academic achievement. 

In addition to the schools mentioned above, schools predominantly or exclusively 
serving over-age students 3 were excluded from the analysis of grade-level retentions, 
grade-level promotions, and the four-year graduation rate. 

Because admissions processes differ by school type, experimental methods could not 
be used to study Pilot high schools (Betts and Hill, 2006). Of the nonexperimental, 
observational methods suggested by Betts and Hill (2006), this paper uses student-level 
analyses of trends in outcomes over time that control for certain individual student 
characteristics, such as race, socioeconomic level, and being a member of a risk factor 
group. The three first-time-ninth-grader risk factors studied in this paper are: low 
attendance in eighth grade, warning on grade 8 math MCAS exam, and being more than 
two years over age. 

All indicators, except for attendance rates, represent the proportion of students within 
each school type who reflect that measure. For example, in terms of out-of-school 
suspensions, the numbers represent the percentage of students within each school type 



2 Students attending the following BPS schools were excluded from the analysis: Carter Center, Community Academy, 
Expulsion Alternative Program, Horace Mann, McKinley Schools, Middle School Academy, and Young Adult Center. 

3 The following schools serve over-age students: Boston Adult Technical Academy (BPS), Boston Day and Evening 
Academy (Pilot), and Greater Egleston Community High School (Pilot). 
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who were suspended in each school year. Median attendance rates were calculated as the 
days students attended divided by days of membership for each school type each year. 

Pilot School group sizes are smaller than BPS and exam schools (Appendix 3 contains 
group sizes for each analysis). Therefore, small changes in the data could result in large 
fluctuations by proportion. Multiple hypotheses were tested to understand fluctuations 
from year to year in Pilot School outcomes, including school openings and closings, 
school size changes, feeder patterns, etc. 

Because the database includes the entire population (students in Boston Public Schools 
from 2001-2005) rather than a random, representative sample, sampling error is 
eliminated and statistical significance testing is not appropriate. Statistical significance 
refers to how certain one can be that a difference seen in a sample can be generalized 
to a population. We use a series of non-parametric tests to make comparisons between 
school types. In addition, in order to determine the strength of the relationship or 
magnitude of difference between the variables in each analysis, effect sizes using the 
Pearson r family of values are reported. Effect sizes can range from -1.0 to +1.0. The 
interpretation of effect sizes in the Pearson r family used the general guidelines of 0.10 
as small, 0.30 as medium, and 0.50 as large (Cohen, 1988). Any effect size is statistically 
meaningful, whether small, medium, or large. 

The Mann-Whitney U test was used to compare BPS and Pilot high schools when the 
dependent variable was ordinal or nonnormally distributed, such as attendance rates, 
out-of-school suspensions, and MCAS exam pass rates. Effect sizes were calculated 
using the formula r = z/a/N. Chi square was used to compare BPS and Pilot high schools 
when the variables were both nominal, such as retention rates, promotion rates, and 
proportion of students with the risk factor of warning on grade 8 math MCAS exam. 
When chi square was used, phi or Cramer’s V provided information about effect sizes. 

More specific methods and descriptions of each indicator, as well as some limitations of 
the study, are included in Appendix 4. 
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Findings 



I. How Are Boston Pilot High School Students Faring? 

Over four years, Pilot high school students on the whole outperform the district average on all 
indicators of student engagement and achievement. 

Because no one indicator can accurately describe a school’s effectiveness, data on 
school and student outcomes cover a range of indicators of student engagement and 
performance. For every indicator examined, Pilot high school outcomes are higher on 
average than non-Pilot, non-exam school outcomes, and differences are statistically 
meaningful. For brevity and clarity, the non-Pilot, non-exam school type is referred 
to as “BPS” throughout this paper. The analyses in this paper build on those reported 
previously (Tung, Ouimette, and Rugen, 2006) and add new indicators made possible by 
multiple years of data. 

Student Engagement 

Student engagement in high school is reflected in multiple ways, both in academics and 
in behavior (Janosz et al., 2000). Several indicators available for analysis of student 
engagement include: 

• Attendance rate 

• Out-of-school suspension rate 

• Rate of transfers out of the school to another school in the district 

• Rate of transfers out of the school to another district 

• Annual drop-out rate 

Attendance 

Attendance rates are an indicator of school effectiveness and correlate with high school 
completion rates (Binkley and Hooper, 1989; Bryk and Thum, 1989; Sween et al., 1987). 
Pilot high school students have consistently high median attendance rates over the 
four years studied, around 94%. The difference between Pilot and BPS attendance rates 
in 2004-05 corresponds to almost two weeks of school. Mann-Whitney U tests were 
performed to compare school types. In all four years, Pilot high schools had higher 
attendance rates than BPS schools, with r ranging from 0.11 to 0.19. 4 



4 Where difference or association has been established through statistical testing, the effect size gives an indication of 
the magnitude of the difference (or association) between variables, in this case, school type and median attendance 
rates. Throughout this study, the interpretation of effect sizes in the Pearson r family uses the general guidelines of 0.10 
as small, 0.30 as medium, and 0.50 as large (Cohen, 1988). Any effect size is statistically meaningful, whether small, 
medium, or large. 
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Figure 1 : Median Attendance Rates by School Type (in percent) 5 
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Out-of-School Suspensions 

Out-of-school suspensions result when students’ behaviors are considered to be 
disruptive. One result of this form of discipline is that the students who are suspended 
are excluded from learning (Cotton, 1995; Pinnell, 1985). Low out-of-school suspension 
rates indicate higher student engagement and positive school climate (Cotton, 1990). 
Over four years, Pilot high schools have had consistently lower suspension rates than 
BPS schools. These rates have also steadily declined over the past two years. Cramer’s V 
tests were performed to compare school types. In all four years, Pilot high schools had 
lower suspension rates than BPS schools, with effect sizes ranging from 0.12 to 0.13. 



Figure 2: Suspension Rates by School Type (in percent) 
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5 In all figures with these terms, “BPS” denotes high school students enrolled in Boston Public Schools who attended 
schools that did not have Pilot or exam status; “BPS Pilot” denotes high school students enrolled in Boston Public 
Schools that have been granted Pilot status by the district; and “Exam” denotes high school students enrolled in the 
three Boston high schools that admit students based on entrance examination results and prior academic achievement. 
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Yearly Transfers within the District, Transfers out of the District, and Annual 
Dropouts 

These indicators are proxies for a school’s mobility rate. High mobility rates indicate 
that the schools are not “holding” students (Rumberger and Thomas, 2000; State 
University of New York, 1992). Students who transfer from a Boston school to a school 
outside of the district and inform the district of the move are categorized as transfers 
out. Pilot transfer-out rates were lower than BPS transfer-out rates for the first three 
years of analysis; however, the rate increased to slightly above BPS rates in the last year 
of analysis. 

Figure 3: Transfer Rates out of District by School Type (in percent) 
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An analysis of transfers out of one Boston school to another Boston school showed that 
Pilot high school in-district transfer rates were the same as BPS high schools for the 
most recent two school years. In-district transfer rates are uniformly low, likely because 
of the district’s voluntary transfer policy, which at the high school level allows only one 
transfer, except for safety and programmatic reasons, during a student’s high school 
career. 



Table 2: In-District Transfer Rates by School Type (in percent) 6 





2002-03 


2003-04 


2004-05 


BPS 


4.3 


5.5 


5.9 


BPS Pilot 


6.7 


4.6 


5.9 


Exam 


3.2 


3.0 


3.1 



6 In all tables with these terms, “BPS” denotes high school students enrolled in Boston Public Schools who attended 
schools that did not have Pilot or exam status; “BPS Pilot” denotes high school students enrolled in Boston Public Schools 
that have been granted Pilot status by the district; and “Exam” denotes high school students enrolled in the three Boston 
high schools that admit students based on entrance examination results and prior academic achievement. 
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Annual drop-out rates are another component of mobility within a student population. 7 
The annual drop-out rate is the proportion of students each year who leave the BPS 
system without transferring and without a diploma. This indicator also includes those 
students for whom the district does not have information on their reason for leaving. 
Pilot high schools have lower annual drop-out rates than BPS each year of the study. 
Cramer’s V for transfers out of district and annual dropouts for the four years ranged 
from 0.11 to 0.14. 

Figure 4: Annual Drop-Out Rates by School Type (in percent) 
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When out-of-district transfers, in-district transfers, and annual dropouts are added 
together, Pilot high schools have lower mobility rates (from 2.0 to 8.5 percentage points 
lower in each year) and higher holding power than other BPS high schools. 

In summary, Pilot high school students had engagement outcomes that were consistent 
and stronger than BPS high school students in attendance, suspensions, and mobility 
as defined by transfers and yearly dropouts. These positive Pilot high school results 
suggest that the schools are engaging students well and predict that their academic 
outcomes will also be positive. 




6.5 



-♦ — BPS 
-■ — BPS Pilot 



Exam 



3.0 



0.4 



0.3 



0.4 



-4 0.4 



7 Annual drop-out rates differ from four-year cohort drop-out rates. Four-year cohort drop-out rates are calculated by 
dividing the number of graduates in a year by the number of ninth graders who entered four years before. 
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Student Performance 



Taken together, student performance indicators provide an explanation of how well 
students progress on time and meet state standards. These academic outcomes are 
analyzed by school type for the district and add to the information that the student 
engagement indicators reveal about high schools in Boston. The indicators described in 
this section include: 

• Grade-level retention rate 

• Grade 9 promotion rate 

• Pass rate in grade 10 ELA and math MCAS exam 

• Changes in math MCAS pass rate from grade 8 to grade 10 

• Four-year cohort graduation rate 

Grade-Level Retentions 

Students who are retained in a grade for an extra school year have an increased risk of 
continued low achievement and of dropping out of school (Jimerson, Anderson, and 
Whipple, 2002). While BPS high schools had high levels of grade retentions for grades 
9-12 (one in four students), Pilot high schools had lower levels of grade retentions, 
although those levels rose in 2003-04 and 2004-05. 

Figure 5: Grade Level Retention Rates by School Type (in percent) 
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To investigate whether BPS and Pilot high schools differ on grade-level retentions in 
grades 9-12, a chi-square statistic was used and differences were found. Effect sizes for 
BPS and Pilot high schools’ grade-level retention rates for all three cohorts ranged from 
0.19 to 0.21 (Cohen, 1988). 
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Grade Nine Promotions 

Grade-level promotion is an indicator of the percent of students who are progressing 
academically at the expected rate. Various studies have shown that many schools are 
promoting ninth graders at lower rates, possibly due to the increased high-stakes 
testing in the tenth grade (Haney et ah, 2004; Warren and Corl, 2007). Because of 
these observations, the promotion rates of ninth graders (as opposed to the previous 
indicator, which combines grades 9-12) were analyzed by school type. Pilot high school 
ninth-grade promotion rates were consistent at 92% across multiple years, approaching 
exam school rates, while BPS high school ninth-grade promotion rates were 16-22 
percentage points lower. A chi-square statistic was used, and Cramer’s V ranged between 
0.20 and 0.27 for the three cohorts. 



Table 3: Grade 9 Promotion Rates by School Type (in percent) 





2001-02 


2002-03 


2003-04 


BPS 


71.5 


69.8 


75.5 


BPS Pilot 


91.7 


92.1 


91.7 


Exam 


93.3 


96.0 


94.2 



MCAS Exam Pass Rates 

MCAS exam outcomes in English/Language Arts (ELA) and math were analyzed by 
pass rates (proportion of students performing in advanced, proficient, and needs 
improvement categories) for four years. In ELA, Pilot students outperformed BPS 
students in each of the four years studied. Exam school students consistently passed at 
rates between 99% and 100% each year. 

Figure 6: Grade 10 ELA MCAS Exam Pass Rates by School Type (in percent) 
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In math, Pilot students outperformed BPS students in three of the four years. 



Figure 7: Grade 10 Math MCAS Exam Pass Rates by School Type (in percent) 
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Mann-Whitney U tests were performed to compare BPS and Pilot high schools. Pilot 
high schools had higher pass rates than BPS schools in all four years for ELA and in 
three out of four years for math. Measures of effect size r ranged from 0.19 to 0.22 for 
ELA and from 0.09 to 0.15 for math. 

Changes in Math MCAS Exam Pass Rates 

Students currently take the mathematics MCAS in grade 8, at the end of middle school, 
and grade 10, after two years of high school. Most students change schools from eighth 
grade to ninth grade, enabling a comparison of tenth-grade MCAS outcomes in one 
school to eighth-grade MCAS outcomes in the previous school two years before. Since 
the goal is to attribute the change in pass rates to the second school, schools that span 
middle and high school were filtered out of this analysis. 8 Using a four-year database, 
it is possible to complete this analysis by school type for two cohorts. The table below 
shows the percentage point changes for two cohorts of students. For the first cohort, 
the change in pass rates from eighth- to tenth-grade tests was higher for Pilot high 
school students than for BPS high school students. For the second cohort, there were 
similar changes in pass rates from the eighth-grade to the tenth-grade tests in BPS and 
Pilot high schools. 9 



8 Boston schools that span middle and high school grades are: the three examination schools and Josiah Quincy Upper 
School (Pilot). 

9 An analogous analysis of changes in ELA MCAS exam pass rates was not possible. ELA MCAS exams were administered 
in grades 7 and 10 during the years of this study, allowing analysis of only one cohort of students. In addition, changes in 
ELA pass rates could be attributed to the middle school (grade 8) and high school (grades 9 and 10). 



Strong Results, High Demand: A Four-Year Study of Boston's Pilot High Schools 



www.cce.org 




